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ABSTRACT 

A study on basic speech assessment replicated an 
earlier one except that in place of the CCAI and the Competent 
Speaker form, the Self-Perceived Public Speaking Competency Scale 
(SPPSC) was used. Also, 2 randomly selected control groups of 
students were added to the research design: one consisting of 62 
students who had not taken speech and were not taking it, and the 
other consisting of 71 students who had already taken speech but were 
not presently enrolled in a speech class. The experimental group 
consisted of 56 students. All three groups were administered the 
PRCA-24, the RSE, and the SPPSC at the beginning and end of the 
semester. Scores were examined in terms of the following hypotheses: 
(1) that all three instruments, pretest and posttest, would not 
register significant differences for the control groups; and (2) that 
all three instruments would indicate better posttest and pretest 
scores for the experimental group. Results showed both hypotheses to 
be wrong. All *3 groups scored lower on the PRCA posttest than the 
pretest; all three groups improved slightly on the other two tests. 

In other words, most students improved in their speech skills over 
time whether enrolled in basic speech or not. The results appear 
disappointing. One explanation would loo*, to the long-proven 
confusion produced by time itself. Another would note that public 
speaking skills were not reinforced in classes other than speech, 
even though speech or presentations were required. (Contains 7 tables 
of scores and 15 references.) (TB) 
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TOWARD REFINING THE ASSESSMENT OF THE BASIC PUBLIC 
SPEAKING COURSE;. AN EXPERIMENTAL STUDY 



All across the country, issues of evaluation and accountability are spurring efforts by 
researchers to define and hone models, methodologies, and instruments for assessing 
skills learned by students in academia including communication skills and competencies 
(Cronin, 1992; Speech Communication Association, 1993; Speech Communication 
Summer Assessment Converence, 1994). Among the proponents of this movement is 
Phil Backlund (1990) who cites two objectives of assessing college outcomes: (1 ) to 
improve the quality of educational programs according to goals that have been set and 
(2) to determine if students achieve these goals. To meet these objectives, Backlund says 
that schools must formulate a statement of anticipated outcomes, participate in college 
assessment programs, and use standardized measures of assessment. 

To address these concerns, communication faculty at a midsize midwestem university 
have been working on developing an assessment program for oral communication skip's 
at the undergraduate level for the past several years. The program those faculty have 
developed has, in part, utilized a pretest-posttest design for the assessment of the 
cognitive, affective, and behavioral dimensions of basic public speaking competency 
(e.g., Morley, Morreale, & Hulbert-Johnson, 1991; Morreale, 1990; Morreale, Hackman, 
Shockley-Zalabak, & Gomez, 1991; Morreale, Moore, Taylor, Surges-Tatum, & Hulbert- 
Johnson, 1993). That design has employed four instruments: Rubin’s (1982) 
Communication Competency Assessment Instrument (CCAl), McCroskey’s (1970) 
Personal Report of Communication Apprehension (PRCA-24), Rosenberg’s (1965) Self- 
Esteem Scale (RSE), and (Morreale, Morley, & Naylor, 1991) the Competent Speaker 
Speech Evaluation Form. 

Recently, a sister institution in a neighboring city, also a midsize university, has joined in 
a collaborative effort to extend and refine the original research design for assessing the 
impact of the public speaking course on undergraduates. (See Campbell, D.G., & 
Morreale, S.P., 1995). Specifically, the second university extended the pre-posttest 
design to include a pre-posttest control group. The intention of this second study was to 
speak to challenges to the first study related to history, maturation, subject selection, etc. 

This paper presents the second study which replicates the first except for the following 
conditions: (1) In place of the CCAl and the Competent Speaker form, Ellis’ (1995) 
Self-Perceived Public Speaking Competency Scale (SPPSC) was used. The scale has 
been reported to be a valid self-report measure of subjects’ perceptions of their 
communication behaviors; and (2) Two randomly selected control groups of students 
were added to the research design: one group consisted of 62 students who had not been 
and were not presently enrolled in a public speaking class; the other group was 
composed of 1 students who had already taken, but were not presently enrolled in a 
public speaking class. The experimental group was composed of 56 students enrolled in 
the basic public speaking course. All three groups were administered the PRCA-24, the 
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RSE, and the SPPSC at the beginning of the semester in January and again at the end of 
the semester in May. 

Scores on the three instruments for all three groups were examined in terms of the 
following hypotheses; 

(1) On all three instruments, pretest-posttest differences for both the non-speech and the 
previous speech control groups were not expected to differ significantly. 

(2) On all three instruments, posttest scores for both control groups were not expected to 
be significantly better than their pretest scores. However, posttest scores for the 
experimental group were expected to be significantly better than the pretest scores. 

Results 

The hypothesis of the experimental group pre-post interaction on the PRCA was not 
supported, F(2, 1 86) = 1 .57, p.= .000. However, there was a significant pre-post main 
effect, F(l, 186) =16.14, p.= .000. Overall, subjects in all three groups scored 
significantly lower (i.e., less apprehensive) on the PRCA at the end of the semester 
(56.48) than at the beginning (59.45). See Table 1 .) 

Analysis of the four PRCA subscales yielded some significant differences. A repeated 
measures ANOVA on the Group Discussion communication apprehension subscale 
yielded a highly significant pre-post main effect, F(l,186) = 9.06, p. = .003, as well as a 
marginal experimental Group Discussion pre-post interaction, F(2, 186) = 2.99, p. = .053 
(see Table 2 for means). Specifically, Group Discussion subscale scores decreased (i.e., 
indicated less apprehension) overall ( 1 5.26 to 14.45). In addition, although only 
marginally significant, it appears that prior speech students remained the same 
throughout the semester while current speech students and those who had never taken the 
course improved somewhat on this dimension. 

Analysis of the Public Speaking subscores of the PRCA yielded results similar to the 
overall PRCA. There was a pre-post main effect, insofar as all three groups improved 
(were less apprehensive) between the beginning and the end of the semester ( 1 8.89 to 
1 7.72), F ( 1 , 1 86) = 1 5.40, p. = .000 (see Table 3). 

A similar pre-post main effect, F( 1 ,1 86) = 8.22, p. = .005, resulted from the Meetings 
subscale pf the PRCA. As the means displayed in Table 4 show, the sample improved as 
a whole over the course of the semester, evidenced by lower scores at the end. 

Analyses yielded no comparable pre-post differences on the Interpersonal Conversations 
subscale of the PRCA (Table 5). 




2 



4 



Analysis of RSE scores yielded results similar to the PRCA. There was a pre-post main 
effect, insofar as all three groups improved between the beginning and the end of the 
semester (40.10 to 41.93), F = 14.18, p. = .000 (see Table 6). 

However, the hypothesis that current speech students would demonstrate more 
improvement than any of the other groups, or that students who had never taken speech 
would score lower overall were not supported, all F’s <1, not significant. 

Analysis of SPPSC scores yielded a significant pre-post main effect, F(l,186) = 6.72, 
p. = .01. Overall, students’ scores increased significantly during the course of the 
semester, regardless of group (66.91 to 68.40). In addition, there was an experimental 
group pre-post interaction, F(12,186) = 4.99, p. = .008. Specifically, prior speech 
students’ scores remained virtually the same. Currently enrolled students increased 
dramatically (63.84 to 68. 12), and those who had never taken speech increased, but less 
so than the current speech students (67.89 to 69.42) (See Table 7.) The students who had 
never taken and were not enrolled in speech class during that semester scored higher on 
the posttest than the other two groups. 



Discussion 



Main effects: 

The results may be summarized as follows: Because pre-posttest main effects were 
found for six of the seven measures used in the study (the PRCA, Group Discussion, 
Public Speaking, Meetings, RSE, and SPPSC), it is obvious that all three of the groups 
reported that they improved over time. So, whether students had taken, had not taken, or 
were enrolled in speech class, they all perceived that they became less apprehensive, 
more competent, and gained higher self-esteem by the end of the semester. Therefore, to 
declare that speech course instruction was most likely the reason that the experimental 
students’ improved in communication skills in this study would be a misnomer. 

Additionally, in all measures except for the SPPSC, currently enrolled students had the 
highest initial concern about their communication competence. This may be accounted 
for by the nature of the course objectives in the speech course (i.e., calling attention to 
such skills) versus non-emphasis of such course objectives in non-speech classes. 

Interaction effects: 

Pre-posttest interaction was marginally obtained on the Group Discussion subset of the 
PRCA, i.e.,. prior speech students’ scores remained the same, while the current speech 
enrollees and the never exposed groups improved only a little. For the SPPSC, pre- 
posttest interaction showed that the prior speech students remain virtually unchanged, 
while those students who had never taken the speech class improved some and the 
current speech enrollees improved the most. 3'he students who had never taken and were 
not enrolled in speech class during that semester scored higher on the both the pre- and 
posttest than the other two groups. 



No effects: 

The seventh assessment, the interpersonal Conversations subscale of the PRCA, was the 
only measure where there were no discemable pre-post differences. Because it is a 
subscale of the PRCA which was found to have main effects overall and for the other 
three components, the findings are not noteworthy. 

The results appear to be disappointing. The numbers show that the public speaking 
experimental group improved their scores on all three instruments at the end of the 
treatment, but so did the two control groups, i.e., those who already had taken a public 
speaking course and those who had not. The obvious contention for such results is the 
pretest, control group, posttest design. Scholars have long noted confounding conditions 
associated with the passage of time, environmental influences, measurement 
sensitization, fatigue, etc., effects which are present in the experimental and the control 
groups (e.g., Kerlinger, 1986; Osgood, Suci, Tannenbaum, 1957). 

However, there may be another explanation. Skills in the art of public speaking that were 
reinforced in the experimental group might have been unnoticed in the other courses. 
Therefore, even though students in both control groups could have made presentations in 
some of the other classes they were taking, their grades might not have included their 
actual speechmaking performance level in front of the group. 

This was certainly the case in at least one class. The author was invited visit to a senior 
level science class where the students presented persuasive speeches with visual aids 
about their semester projects/inventions to the author. It was noted that all but one of the 
approximately 20 seniors in the class had taken the public speaking class as a freshman 
or junior. Almost all of the presentations critiqued by this author in that class had at least 
some important public speaking drawbacks, even after a refresher module presented by 
the author to that group the month before their presentations. Had a potential marketer or 
investor in the business world been in attendance, the presentational skills of these 
college seniors might have influenced that business person’s interest in the products 
presented. Similar performance levels have been noted in the author's senior level 
classes as well. Even former students of the author’s speech class seem to have forgotten 
their speechmaking skills when presenting work in the author's senior level classes. 

Maybe the problem is not that speechmaking skills weren't learned, but that they are not 
being reinforced in later coursework. A student who takes the speech course as a 
freshman or sophomore might forget such skills because they are not reinforced in 
successive classes. Another difficulty is that sometimes instructors do not realize what is 
missing in the presentation, sometimes possibly assuming that people can present 
information effectively through sheer experience rather than training. 

Therefore, recommendations for future study are twofold. The first is insuring that the 
basic spteech course is included in the curriculum for all majors, whatever the field, and 
reinforcing those skills in later coursework to assure not only that the student learns, but 
that the student does not lose a precious con-;petence in what it takes to succeed in life 
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after college. The second recommendation is to continue to look for appropriate 
measures of the outcome variables. Problems inherent in the pre-posttest design might 
yet be overcome through future development of better measures. 
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Table 1: Pre-Post Overall PRCA Scores 





Prior 


Current 


Never 


Mean 


Pre 


58.18 


62.11 


58.50 


59.45 


Post 


56.87 


57.55 


55.05 


56.48 . 






Table 2: Pre-Post Group Discussion Scores 




Prior 


Current 


Never 


Mean 


Pre 


14.48 


16.32 


15.21 


15.26 


Post 


14.58 


14.88 


13.92 


14.45 



Table 3: Pre-Post Public Speaking Scores 





Prior 


Current 


Never 


Mean 


Pre 


18.83 


19.39 


18.52 


18.91 


Post 


18.07 


17.73 


17.47 


17.76 



Table 4: Pre-Post Meeting Scores 





Prior 


Current 


Never 


Mean 


Pre 


15.94 


16.96 


15.68 


16.16 


Post 


15.73 


15.66 


14.66 


15.36 





Table 5: 




Prior 




Prior 


Pre 


8.93 


Post 


8.49 
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Pre-Post Interpersonal Conversations Scores 
Current Never Mean 

Current Never Mean 

9.43 9.10 9.13 

9.00 

8 10 



9.23 



8.88 



Table 6: Pre-Post RSE Scores 





Prior 


Current 


Never 


Mean 


Pre 


40.04 


40.21 


40.07 


40.10 


Post 


41.27 


41.96 


42.66 


41.93 



Table 7: Pre-Post SPPSC Scores 





Prior 


Current 


Never 


Mean 


Pre 


68.48 


63.84 


67.89 


66.91 


Post 


67.73 


68.12 


69.42 


68.40 
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